SPECIFICATION 



TITLE OF INVENTION: 

"EnsoBox Clustered Services Architecture: Techniques for enabling the creation of a scalable, robust, 
and industrial strength Internet services provider appliance" 

CROSS-REFERENCE TO RELATED APPLICATIONS 

Not Applicable 

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR 
DEVELOPMENT 

Not Applicable 

REFERENCE TO SEQUENCE LISTING, A TABLE, OR A COMPUTER PROGRAM 
LISTING COMPACT DISK APPENDIX 

Not Applicable 

BACKGROUND OF THE INVENTION 

TECHNICAL FIELD OF INVENTION: 

Clustered Services Architecture describes a configuration of computing devices that allow for the 
delivery of reliable services to Internet users. This configuration deals with the problems of service 
scalability, performance, redundancy and fault tolerance. This configuration works with computing 
technologies such as virtual local area networking, network attached storage, distributed computing and 
clustering of services. 
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The Problem: Building the best architecture to deliver Internet services 

An Internet Service Provider (ISP) delivers Internet services to businesses and consumers. Internet 
services are applications that are offered to users over the Internet. These services include, but are not 
limited to, dial-up IP, email, chat, portal, electronic commerce and news. When building architecture to 
deliver these Internet services, the following issues arise: 

It is difficult to scale the infrastructure to support growing user bases and additional Internet 
services. 

Services rely on computer hardware and computer hardware fails quite often, creating costly 
downtime. 

Due to the nature of Internet services and increasing savvy of Internet users, security is difficult to 
manage. 

Finding technical talent capable of managing the system resources is increasingly difficult. 
Two Typical Approaches 
Enterprise Solutions 

The use of enterprise class hardware and software is becoming the standard for building architecture 
to support Internet services. The term enterprise typically refers to larger corporations, so enterprise 
hardware and software is geared toward larger corporations. 

Enterprise platforms have a number of features that benefit the service provider market. Enterprise 
hardware platforms scale easily. Additional modules that carry additional capacity or functionality can 
be added to most enterprise hardware platforms. Due to technology advances such as hot swapping 
hardware, it becomes practical to scale the enterprise hardware platform. Also, redundancy of 
hardware minimizes potential system downtime. 

Enterprise software platforms are deployed on enterprise hardware platforms. They rely on the 
enterprise hardware for scale. Enterprise software platforms can be tuned and optimized for these 
hardware platforms. Various enterprise hardware and software must be tightly integrated to provide 
architecture capable of delivering Internet services. 



The largest benefit to using enterprise class hardware and software are the corporate alliances that 
the enterprise solution providers bring with them. These alliances bring network, storage, server and 
software companies together to provide a solution for vertical market segments. There are several 
enterprise solutions for the service provider market. 

Figure 1, provides a typical example of an enterprise solution geared towards the service provider 
market. The architecture described is capable of providing Internet services in a reliable manner. The 
servers are standard enterprise servers clustered together with software. The storage is arranged in a 
standard Storage Area Networks (SAN) design. The router and switch are assumed fault-tolerant and 
could be duplicated for redundancy. This architecture (hardware and software) can be purchased 
within a six hundred thousand dollar price point. 

It would appear that an enterprise solution would be the best option for delivering Internet services in 
a reliable manner; however, it is an extremely expensive option. 

Due to extreme cost concerns, it is difficult for the service provider industry to make use of enterprise 
solutions. Service providers require hardware and software solutions that grow incrementally so that 
the cost of their architecture will only be incurred as their customer base and revenues grow. Due to 
these cost concerns, a completely new architecture should be built so that the service provider can 
grow incrementally. 

Teamwork Approach 

Due to the price of enterprise solutions, service providers have turned to integrating existing low-end 
hardware and software in order to deliver reliable Internet services. 

Service providers have recognized that enterprise solutions are overkill for their needs. The margin 
on low-end hardware and software is much less than enterprise hardware and software, so the price 
point for low-end hardware and software is decreasing daily. This makes low-end hardware and 
software integration appear to be a lucrative option for the service provider market. 

The teamwork approach to creating architecture capable of providing Internet services is extremely 
labor intensive. A strong team of engineers can integrate low-end hardware in order to provide 
architecture for a service provider. They can build a system to provide reliable Internet services by 
using a combination of inexpensive servers, storage and network devices. 



Scaling is simply a matter of adding more low-end hardware to the mix. The team develops methods 
for capacity planning and performance monitoring in order to incrementally scale the architecture. 
They also devise a monitoring procedure and will respond quickly to system failures. 

A strong team of engineers can also integrate existing custom off the shelf (COTS) software in order 
to deliver reliable Internet services. There is a variety of low-end software that will provide Internet 
services. In fact, much of this software is available from various public domain sources. This software 
can be integrated with external scripts and programs written by the service provider's engineering 
staff in order to provide a full suite of Internet services. 

While there is a large savings from utilizing low-end hardware and software, the teamwork approach 
is not without significant costs. This approach is extremely labor intensive. Good engineering talent is 
very expensive in today's technical market place. 

Since the individual technology companies have not formed alliances to deliver lower-end solutions, 
there is no standard for this approach. Due to the lack of standardization, greater effort is required to 
design, deploy, and administer this architecture. As this type of architecture grows, the engineering 
staff resource requirements, and cost of those resources will grow exponentially. 

Summary of Two Typical Approaches 

Enterprise solutions and teamwork solutions are both good approaches for deploying infrastructure to 
support service provider offerings. However, each option is not without costly downsides. 

Enterprise solutions bring best of breed hardware and software together in a standard way to provide 
Internet services. This standardization enables ease of scale and administration thus limiting the 
necessity for large amounts of engineering staff resources. However, enterprise solutions are extremely 
pricey due to the high cost of enterprise class hardware and software. 

The teamwork solution makes use of the lower end hardware and software, but it requires large 
amounts of engineering staff resources due to the lack of corporate alliances and standardization. 
Since engineering staff is quite expensive and difficult to obtain, this option will be extremely pricey and 
very time consuming. 

Figure 2, summarizes the individual PROS and CONS of each approach. 

The key to leveraging the advantages of each approach, is standardization. Lower-end hardware and 
software can be integrated to accomplish the same end goal as standard enterprise solutions. 
However, standardization is critical to create ease of scale and administration and to reduce costs by 



limiting the need for large amounts of technical staff. A company should take the initiative to create a 
standard platform that integrates the lower-end hardware and software from best of breed hardware 
and software manufacturers. This integration should be geared towards a standard platform for the 
service provider industry. 

BRIEF SUMMARY OF THE INVENTION: 

Clustered Services Architecture (CSA) defines a standard architecture for the service provider market. 
This approach integrates best of breed hardware and software to provide an architecture capable of 
delivering Internet services in a reliable manner. CSA combines the benefits of enterprise solutions and 
the teamwork approach in a much more cost efficient way. The CSA: 

scales independently and incrementally as the customer base and service offerings grow. 

uses lower-end hardware and software from best of breed hardware and software manufacturers. 

and is designed to fit the specific needs of the service provider market. 

At the heart of CSA are two methods, each with important benefits: the Well Defined Architecture and 
Services Clustering. The Well Defined Architecture standardizes the hardware in CSA. It is a standard 
architecture for integrating lower-end hardware from best of breed hardware manufacturers for service 
providers. This architecture uses open system standards throughout the integration process, ensuring 
that there is no dependency on a particular hardware vendor. It is possible to use any hardware vendor 
that adheres to the open standard. 

The second method is known as Services Clustering. Services Clustering defines the software 
architecture component of CSA. This method sets standards for the deployment of the back-end and 
front-end software required to enable Internet services. This method groups services together to make 
efficient use of the standard hardware architecture (Well Defined Architecture). 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING: 

For a fuller understanding of the present invention, reference is made to the following detailed 
description taken in conjunction with the accompanying drawing figures wherein like reference 
character denote corresponding parts throughout the several views and wherein: 

FIG. 1 is an example of a typical enterprise architecture; 

FIG. 2 is a summary of the PROs and CONs of existing approaches to problem; 

FIG. 3 describes the three layers of the well-defined architecture; 



FIG. 4 details the difference between services clustering and non-clustering; 
DETAILED DESCRIPTION OF THE INVENTION: 

Clustered Services Architecture (CSA) defines a standard architecture for the service provider market. 
This approach integrates best of breed hardware and software to provide an architecture capable of 
delivering Internet services in a reliable manner. CSA combines the benefits of enterprise solutions and 
the teamwork approach in a much more cost efficient way. The CSA: 

scales independently and incrementally as the customer base and service offerings grow. 

uses lower-end hardware and software from best of breed hardware and software manufacturers. 

and is designed to fit the specific needs of the service provider market. 

At the heart of CSA are two methods, each with important benefits: the Well Defined Architecture and 
Services Clustering. The Well Defined Architecture standardizes the hardware in CSA. It is a standard 
architecture for integrating lower-end hardware from best of breed hardware manufacturers for service 
providers. This architecture uses open system standards throughout the integration process, ensuring 
that there is no dependency on a particular hardware vendor. It is possible to use any hardware vendor 
that adheres to the open standard. 

The second method is known as Services Clustering. Services Clustering defines the software 
architecture component of CSA. This method sets standards for the deployment of the back-end and 
front-end software required to enable Internet services. This method groups services together to make 
efficient use of the standard hardware architecture (Well Defined Architecture). 

Well Defined Architecture 

The Well Defined Architecture standardizes the three layers of hardware necessary to deliver reliable 
Internet services. Each hardware layer works together to provide a complete hardware architecture for 
service providers. The three layers are: 

Network Layer - a combination of network resources that enables the customer to access the 
services (i.e. switches and routers). 

Server Layer - a combination of server equipment that serves the services to the customer (i.e. 
Unix servers). 

Storage Layer - a combination of disk resources that stores information that the services require. 



The well-defined architecture is designed to make efficient use of the system resources. This 
architecture makes use of proven technology to tightly integrate the three tiers. Each tier can scale 
independently and operate efficiently. 

5 Network Layer 

The Well Defined Architecture standardizes the Network Layer with two technologies. 

Virtual Local Area Networks (VLAN) are used at Open System Interconnection (OSi) Layer 2 and 3 
(Data Link and Network) in order to make efficient use of the network resources. 

10 

Switching on a network inherently speeds up network performance. 

VLAN's allow for an inexpensive, flexible way to modify logical groups in changing 
environments. 

□ VLAN's operate independently of the physical medium; which will allow the network to be 

? | upgraded with no impact to the existing Internet services. 

SI VLAN's also add additional data security to the Network Tier. 

i ?s 

; || Load Direction and Content Switching is used at OSI Layer 2,3,5 and 7 (Data Link, Network, Session 

Ljl and Application). Load Direction is used to direct Internet service users to one server from a pool of 

20 servers. Content switching: 

ff : allows the users to be directed to the most available server in the pool. 

enables persistence such that the user is continuously directed back to the server initially 
chosen for a finite period of time. 

25 

Server Tier 

CSA approaches the Server Tier with a technology called ensoRAIS (ensoport.com, Inc.'s Redundant 
Array of Inexpensive Servers). This technology is based on using a group of inexpensive servers as 
opposed to a small number of enterprise class servers. Each server has an exact replica of the 
30 software required to host a service. Because of this, users can access any of the servers and utilize 

the Internet services. Each server in the ensoRAIS design is configured exactly like every other 
server. In the event that more server resources are necessary, it is easy to add another server to the 
configuration without impacting the existing traffic on the current servers. 

35 Storage Tier 

CSA standardizes the Storage Tier with Network Attached Storage (NAS) technology. Network 
Attached Storage introduces a high-performance, highly reliable device that is dedicated to providing 
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storage over a network. NAS devices have a single purpose operating system that reduces the 
overhead of reading and writing data from the storage medium. Adding more disks can scale these 
devices. They operate independently of the server that requires the information that they store. 

Within CSA, the server layer accesses information from the NAS devices using Network File System 
(NFS). NFS maintains file integrity so that files will not be corrupted when multiple machines attempt 
to access them at the same time. 

Figure 3, provides a graphical representation of hardware that used at each of the three hardware 
layers. 

Services Clustering 

While the Well Defined Architecture standardizes the hardware requirement, Services Clustering 
standardizes the software requirement. This method "clusters" the services, that is to say, it groups the 
services together to make the most efficient use of the standard architecture (The Well Defined 
Architecture). The key to Services Clustering is the Service Package and Service Management. 

Service Package 

The service package is one copy of all of the software required to deliver every Internet service 
offered by the service provider. Each Internet service will require one or more pieces of software in 
order to operate. For example Internet email will require a mail transport agent software, a message 
access software and a web software. Similarly, Internet Chat will require a back-end chat daemon 
software and a front-end chat client. Using CSA, we create an Internet service package that includes 
all of the software required for email and all of the software required for chat. We then load this 
package onto every machine at the Server Layer of the Well Defined Architecture. 

The Content Switching at the Network Layer of the Well Defined Architecture directs users to the 
least loaded machine running the service package. For example, if a user wanted to use Internet 
Chat, they would be directed to the least loaded machine at the Service Layer. Because the software 
required to use chat is loaded on every machine at the Server Layer, every machine is capable of 
providing the service. 

Figure 4, details the difference between using Services Clustering to offer Internet services and a 
non-clustered approach. 



Service Management 



Service Management maintains independent server configurations and automatically installs updates 
(i.e. Security Patches, Bug Fixes, etc..) all software included in the Service Package. This is 
accomplished through the use of remote software distribution. A gold copy of the service package is 
maintained, updated and tested. After testing in a vanilla environment, updates can be easily pushed 
5 to the Server Layer of the Well Defined Architecture. 

Creating a package of all software required to run all Internet services offered by the service provider 
is not difficult, but the configuration of each software package must differ slightly on each machine. 
For example, each server will have a different hostname and that hostname must be stored in a 
10 configuration file. Service Management resolves this issue by maintaining configuration files for all 

software on a machine independent basis. By maintaining a simple database of configuration files, 
changes to configuration files can be tracked and updated automatically. 

Benefits 

tA CSA is an extremely efficient model designed primarily for the service provider market. The primary 
i fe fi benefits gained from CSA include cost savings, staff resource reduction, scalability, and fault tolerance. 

; H Cost Savings 

Lfl The use of CSA standard by a service provider will greatly reduce the initial investment in 

2p infrastructure hardware. Because of the Well Defined Architecture, the service provider is able to 

|*„fc purchase only the computer hardware needed to begin offering Internet services. The service 

CI provider can then incrementally add additional hardware to each layer of the Well Defined 

\ sz x. 

; H Architecture as the hardware is needed. By incurring infrastructure cost incrementally, the service 
q provider will be able to build profits faster and turn more resources back into growing the business 
and enhancing value to the customer. 

Staffing Resource Reduction 

Standardizing hardware (Well Defined Architecture) and utilizing Service Management (Services 
Clustering), have the benefit of minimizing technical staffing requirements. 

30 

Overall administration is made easier because all hardware at each layer of the Well Defined 
Architecture is configured in exactly the same manner, Service Management keeps configuration 
differences and changes in a centralized location for easy administration. It also allows for a 
centralized software upgrade strategy. By making overall administration easy, the service provider 
35 can shift resources and staff to value added activities and positions such as sales and customer care. 

Scalability 
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